Mobile robotic assistant for multipurpose applications

ABSTRACT

Embodiments of the present invention relate to a robotic assistant comprising a projector for projecting media on a surface, a sensor for sensing the media, and a motion control module for moving the robotic assistant.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention generally relate to mediaprojection, and more particularly, to a mobile robotic assistant formultipurpose applications.

2. Description of the Related Art

Mobile projectors are often used in home and office settings forprojecting video, presentations, reports and the like to largeaudiences. Often, these projectors are rather large and unwieldy due toincreased circuitry for high definition projection. If the projector isneeded in another area of the house or office, a user must pick it up torelocate it manually. The weight of the projector and the multitude ofcable couplings from the projector make it especially difficult for theelderly or children to move the projector unless they have assistance.In other instances, an office or conference room only has one projectionscreen statically fixed to the ceiling or a wall, available for viewingthe presentation. A problem exists when the user desires that thepresentation or report be shown in a different office or conferenceroom, it becomes difficult to remove all of the cable couplings from theprojector and lift the projector to relocate it. Another problem existswhen a user would like to project media from a projector in an areawhere there are flat surfaces, but calibrating the projector to properlydisplay the media is cumbersome and difficult, involving excessive trialand error by the user leading to user frustration.

Therefore, there is a need in the art for a mobile robotic assistant forassisting in projecting media in different offices and rooms andcalibrating the projections dynamically.

SUMMARY OF THE INVENTION

Embodiments of the present invention comprise a robotic assistantcomprising a projector for projecting media on a surface, a sensor forsensing the media, and motion control for moving the robotic assistant.

Embodiments of the present invention further comprise utilizing arobotic assistant for projecting media on a surface, sensing media in asurrounding environment and moving the robotic assistant.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a functional block diagram representation of a roboticassistant for multipurpose applications, in accordance with exemplaryembodiments of the present invention;

FIG. 2 is a block diagram of a computer system implementation of therobotic assistant, in accordance with exemplary embodiments of thepresent invention;

FIG. 3 depicts a flow diagram of a method for projecting visual media asperformed by the robotic assistant, according to one or more embodimentsof the present invention;

FIG. 4 depicts a flow diagram of a method for executing audio commandsby the robotic assistant, according to one or more embodiments of thepresent invention; and

FIG. 5 depicts a flow diagram of a method for enabling motion of therobotic assistant by audio command, according to one or more embodimentsof the present invention.

DETAILED DESCRIPTION OF EMBODIMENTS

Embodiments of the present invention generally relate to a mobilerobotic assistant for multipurpose applications. The robotic assistantpossesses a plurality of built in functional capabilities including, butnot limited to, audio sensing, with integrated voice commandrecognition, image (or video) sensing, and image (or video) projection.The robotic assistant comprises a projector for projecting mediaincluding, but not limited to, audio and video, a sensor for sensing ordetecting the media and a motion control module for managing movement ofthe robotic assistant.

FIG. 1 depicts a functional block diagram representation of a roboticassistant 100 for multipurpose applications, in accordance withexemplary embodiments of the present invention. By way of example, andin no way limiting the scope of the invention, the robotic assistant 100may be a modular, semi-autonomous robot with a microphone, camera andprojector.

By way of example, and in no way limiting the scope of the invention,the robotic assistant 100 comprises a media output module 102, aninput/output (I/O) interface 104, a media storage module 106, a surfacedetection module 107, a surface detector 108, memory 109, a mediaprojection unit 110, an audio capture and command module 114, a videocapture module 116, a motion control module 120, and motion actuators118.

The I/O interface 104 is configured to couple with various sensingdevices, data ports and the like that are external to the roboticassistant 100. According to some embodiments, I/O interface 104 coupleswith a universal serial bus (USB) port in the form of mini and micro USBinterfaces. In certain scenarios, one or more input devices, such as amicro USB, micro HDMI and VGA, may be coupled through at least one ofone or more analog and digital audio and video connectors and one ormore USB data ports to the I/O interface 104 of the robotic assistant100. One or more network devices, such as WiFi and Bluetooth® devicesmay be coupled through one or more USB data ports to the I/O interface104 of the robotic assistant 100. In one embodiment, the I/O interface104 is coupled to a memory card reader for transferring data fromdetachable storage media. In another embodiment, media is downloadedfrom the Internet and transferred via the I/O interface 104 for storagein the memory 109 of the robotic assistant 100. The I/O interface 104may also be coupled with a visual sensor 105 for sensing a sequence ofimages, such as video. The video capture module 116 is used to capturethe sensed video.

A user of the robotic assistant 100 may capture images with theirpersonal phone or camera and insert the storage medium from that phoneor camera into the memory card reader coupled to the I/O interface 104of the robotic assistant 100. The media storage module 106 reads datafrom the I/O interface 104 coupled to the storage medium and transfersthe data into the memory 109 of the robotic assistant 100. In oneembodiment, there is a filter in the media storage module 106 so thatonly particular types of files are transferred from the storage mediumto the memory 109. In this embodiment, the types of files transferredmay be, for example, moving picture experts group (MPEG) 3/4 files,Joint Photographic Export Group (JPEG) files, Microsoft® Office® files,Audio Video Interleave (AVI) files and the like.

The surface detection module 107 is coupled with the I/O interface 104.The surface detection module 107 uses a surface detector 108 to scansensed video from the visual sensor 105 to detect whether a surface ispresent in the video. The surface detector 108 interacts with the videocapture module 116 and detects edges and corners in the video of asurface. In general, surface detection techniques are well-known tothose of ordinary skill in the art. The surface detection module 107estimates the dimensions and geometry of the projected region in thesurface and sizes the media according to predetermined initialconfiguration parameters. In addition, the surface detection module 107detects the optimal inclination angle and distance from the projectedregion in the surface, signaling the motion control module 120 to alterany one of the dimensions and geometry of, and inclination angle anddistance from, the projection region. These calibrations are performeddynamically and independent of user input by the robotic assistant 100.However, the user may interact with the calibration if desired, andmodify the initial parameters of the configuration.

The surface detection module 107 signals the motion control module 120to have the robotic assistant 100 move both horizontally and verticallyin iterations so as to fit the media in the projection region of thesurface with proper proportions according to the configurationparameters, as well as automatically adjusting the projection forclarity and sharpness, in response to voice commands generated from theaudio capture and command module 114. In addition, the surface detectionmodule 107 signals the motion control module 120 to modify theinclination angle of the robotic assistant 100.

The video capture module 116 may also capture video at the direction ofa user and store that video to memory 109 using the media storage module106 of the robotic assistant 100. In addition, the audio capture andcommand module 114 also stores audio in memory 109 sensed by a couplingto the I/O interface 104, preferably through a microphone or other audiosensor. The audio capture and command module 114 captures audio and usesthe media storage module 106 to store the audio to memory 109. The audiocapture and command module 114 later may determine that the capturedaudio is an audio command to perform an action by the robotic assistant100.

In one example, an audio command is operable to engage the motioncontrol module 120. The motion control module 120 determines the type ofmotion the robotic assistant 100 will engage in by activating the motionactuators 118. In an exemplary embodiment, the motion actuators 118 areindividual wheels, coupled wheels, Caterpillar® tracks or the like andthe motion control module 120 accepts motion instructions in the form ofcoordinates or other spatial or direction and speed identifiers.

The media projection unit 110 is a device that, according to oneembodiment, projects an image, a video, a presentation or any visualsignal onto any surface, and commonly a projection screen. The mediaprojection unit 110 can be used to project contents of files from thememory 109, which may be downloaded from the internet transferred fromdetachable media, or captured by the video capture module 116 of therobotic assistant 100. For example, if a user wants to project a paperdocument or an image of a prototype, in an exemplary embodiment thevideo capture module 116 captures video of the document or prototype andthe media output module 102 interacts with the I/O interface 104 to senda signal to the media projection unit 110. The media projection unit 110projects the signal onto a projection screen. In certain specificembodiments, the I/O interface 104 is used to receive data allowing therobotic assistant 100 to clone an external display, thereby facilitatingeasy viewing of images from the external display in a larger form. Anexternal display is cloned when the media output module 102 detects aninput video signal coupled to the I/O interface 104. The Media outputmodule 102 then interacts directly with the I/O interface 104 to havethe media projection unit 110 project the content of the externaldisplay onto a surface.

In exemplary embodiments, the robotic assistant 100 projects onto awall-like surface using the media projection unit 110. The roboticassistant 100 accesses and retrieves the dimensional specifications,i.e. length and width, of the projection area. The robotic assistant 100fits the images or videos to be displayed onto the projected area (orscreen) with proper proportions.

In some embodiments, the video capture module 116 senses the projectedarea and treats the projection like a touch screen. For example, a usercan perform gestures on the projected area and the projected image zoomsout, zooms in, or rotates and translates according to the gesture. Thevideo capture model 116 captures touch points relative to the projectionsurface, digitizes the touch points and translates the digitized pointsinto matrix transforms applicable to the projected view. The mediaoutput module 102 re-displays the altered projection.

In some embodiments, the media projection unit 110 is detachable fromthe robotic assistant 100. For example, the media projection unit 110can plug in to the robotic assistant 100, or can be added or mountedupon the robotic assistant 100. Thus, the projector 110 can be easilymounted on the robotic assistant 100 and carried around at all times, ordismounted and removed as needed.

Although in the above described exemplary embodiments the mediaprojection unit 110 projects a two dimensional (2D) image, the inventionis not limited thereto, and includes in other embodiments a projectorthat projects a signal that gives the viewer the perception of seeing athree dimensional (3D), holographic, image, for example by projectingtwo 2D images that are offset from each other so as to be displayedseparately to the left and right eye of the viewer. In such embodimentswhere a holographic image is projected, surface detection module 107would still sense a 2D surface upon which the 2D images are projected.In other embodiments a holographic image is projected onto a 3D displaysurface.

In even further embodiments the media projection unit 110 may include atransmitter, so that when the media projection unit 110 is used toproject the image signal, instead of optical projection, the signal istransmitted electro-magnetically, typically wirelessly, to a remotedisplay. The remote display is positioned such that a display surfacethereof facilitates easy viewing of the desired media. Thus, as used insome embodiments, the word “projecting” also includes “transmitting”

In official use case scenarios, the robotic assistant 100 is used formultiple purposes including, but not limited to, capturing images andvideos, multimedia projection, voice and/or video call management anddictations or transcriptions. For example, as an office assistant, themedia projection unit 110 of the robotic assistant 100 can be used withfeatures, such as dictations of meetings, video and/or voice callmanagement, and capturing pictures of material presented in the meetingsfor future display.

FIG. 2 is a block diagram of a computer system 200 in accordance withexemplary embodiments of the present invention. The computer system 200includes a processor 202, a memory 204 and various support circuits 206.The processor 202 may include one or more microprocessors known in theart, and/or dedicated function processors such as field programmablegate arrays programmed to perform dedicated processing functions. Thesupport circuits 206 for the processor 202 include microcontrollers,application specific integrated circuits (ASIC), cache, power supplies,clock circuits, data registers, I/O interface 207, and the like. The I/Ointerface 207 may be directly coupled to the memory 204 or coupledthrough the supporting circuits 206. The I/O interface 207 may also beconfigured for communication with input devices and/or output devices,such as, network devices, various storage devices, mouse, keyboard,displays, sensors and the like, collectively referred to hereinafter asI/O devices 208.

The memory 204 stores non-transient processor-executable instructionsand/or data that may be executed by and/or used by the processor 202.These processor-executable instructions may comprise firmware, software,and the like, or some combination thereof. Modules havingprocessor-executable instructions that are stored in the memory 204comprise the media storage module 210, the media output module 212, thevideo capture module 214, the audio capture module 216, the motioncontrol module 218 and the surface detection module 220. In an exemplaryembodiment, the memory 204 may include one or more of the following:random access memory, read only memory, magneto-resistive read/writememory, optical read/write memory, cache memory, magnetic read/writememory, and the like, as well as signal-bearing media, not includingnon-transitory signals such as carrier waves and the like.

According to exemplary embodiments of the present invention, thesoftware modules stored in memory 204 are software implementations ofthe media storage module 106, the media output module 102, the videocapture module 116, the audio capture module 114, the motion controlmodule 120 and the surface detection module 107 of FIG. 1. The motioncontrol module 218 controls the motion actuators 118 and the othersoftware modules send and receive signals to and from the I/O devices208 through the I/O interface 207. In an exemplary embodiment, the mediaprojection unit 110 of FIG. 1 is coupled to the system 200 as one of theI/O devices 208. The motion actuators 118 are also implemented as one ofthe I/O devices 208 and are controlled by the motion control module 218to drive the robotic assistant 100 to move to a particularly specifiedlocation by a user of the assistant 100.

FIG. 3 depicts a flow diagram of a method 300 for projecting visualmedia as performed by the robotic assistant 100 of FIG. 1, according toone or more embodiments of the present invention. The method 300represents execution of the robotic assistant 100 of FIG. 1, implementedas the computer system 200 of FIG. 2, according to an embodiment of theinvention. The method 300 begins at step 302, and proceeds to step 304.

At step 304, the method 300 determines whether a detachable storagemedium is detected at the I/O interface 104. The method moves to step306 if no detachable storage medium is detected. At step 306, the methoddetermines whether video capture is initiated by the user of the system200. In this embodiment, a user can initiate a video capture similar tocamera operation and store the captured video for later use.

In method 300, if video capture is not initiated by the video capturemodule 214, the method 300 proceeds to step 308 where it is determinedwhether audio input is detected by the audio capture module 216. If nostorage medium is detected and video capture is not initiated, and audiois not detected, the method ends at step 320. If detachable storagemedia is detected, or video capture is initiated, or audio is detected,the method proceeds to step 310.

At step 310, data from the storage media, the video capture or audiocapture is transferred to the memory 204 of the system 200. At step 213,the method 300 determines whether the data stored is video data. If thedata is not video data, the method transfers control to method 400described below, shown as “A” in FIG. 3.

If the data stored in memory 204 is video data, the surface detectionmodule 220 performs surface detection on the video data. At step 316, ifa display surface is not detected, the method returns to step 314 toperform surface detection. The surface detection module 220 searches forsuitable surfaces for receiving the projected signal, include, forexample without limitation, flat surfaces such as walls, desks andprojection screens free of clutter, so that it may indicate to the mediaoutput module 212 that a proper surface is available. The roboticassistant 100 accesses the dimensions and geometry of the projected areaon the screen. During the method 300, the robotic assistant 100 can moveabout within an area until the projection appears properly proportionedfor suitable viewing on a projection surface.

If it has been determined that a display surface is detected at step316, the method 300 moves to step 318, where the stored video data(media) from memory 204 is prepared for output by the media outputmodule 212. The media output module 212 interacts with the media storagemodule 210 to retrieve the media from memory 204. The media outputmodule 212 transfers the media to the media projection unit 110 of FIG.1 through the I/O interface 207 for display. The method ends at step320.

FIG. 4 depicts a flow diagram of a method 400 for executing audiocommands by the robotic assistant 100 of FIG. 1, according to one ormore embodiments of the present invention. The method 400 representsexecution of the audio capture and command module 114 implemented as theaudio capture module 216 and executed by the computer system 200 usingprocessor 202 of FIG. 2, according to an embodiment of the invention.

The method 400 begins at step 402, and proceeds to step 404. At step404, the audio control module 216 parses the audio data stored in memory204 of the system 200. The audio data may be simply a voice recordingfor accompanying a presentation, or may comprise complex voice commandsfor moving the robotic assistant 100, playing a video presentation, orinitializing and shutting down the robotic assistant 100 and the like.

The method then proceeds to step 406, where the audio capture module 218interprets the command portion of the audio data as a command for therobotic assistant 100 as described above. At step 408, the commandmodule 218 executes the command and various ones of the above describedmodules of FIGS. 1 and 2 allow the robotic assistant 100 to perform theexecuted command. The method ends at step 410.

FIG. 5 depicts a flow diagram of a method 500 for enabling motion of therobotic assistant 100 by audio command, according to one or moreembodiments of the present invention. The method 500 representsexecution of the audio capture and command module 114, the motioncontrol module 120 and the motion actuators 118 implemented as the audiocapture module 216 and the motion control module 218, executed by thecomputer system 200 using processor 202 of FIG. 2, according to anembodiment of the invention.

The method 500 begins at step 502, and proceeds to step 504. At step504, the command module 217 interprets the audio stored in memory 204 ofthe system 200 as a motion command. The audio data may comprise complexvoice commands for moving the robotic assistant 100, playing a videopresentation, or initializing and shutting down the robotic assistantand the like. In this embodiment, the audio command is a motion commandfor the robotic assistant 100.

The method then proceeds to step 506, where the command module 217 plansthe motion of the robotic assistant 100 taking into account variousobstacles in its path using well known methods for path finding. Themotion command may be a rotation, a translation, or a combination of aplurality of rotations and translations, though the robotic assistant100 is not limited to only these motions. In other embodiments, therobotic assistant 100 can tilt and adjust the media projection unit 110to display on a particular surface such as the ceiling, or on a tiltedsurface.

Once the motion is fully planned by the command module 217, the commandmodule 217 signals the motion control module 218. The motion controlmodule 218 controls the motion actuators 118 through the I/O interface207. The motion actuators 118 perform the planned motion for the roboticassistant 100. The method ends at step 510.

FIG. 6 is an illustration of an implementation of the robotic assistant100 as apparatus 600 in accordance with embodiments of the presentinvention. The apparatus 600 comprises an enclosure 602, a mediaprojection unit 604, a visual sensor 606, an audio sensor 608, an I/Oreader 610 and motion actuators 612. In this embodiment, the motionactuators 612 are a set of two wheeled tracks positioned at oppositesides of enclosure 602, driven by a motor (not shown) so as to moveenclosure 602 transversally along the wheel-shaft 616.

The media projection unit 604 is mounted at the top of the apparatus 600in this embodiment, but may be at other locations in other embodiments.The media projection unit 604 may have an electronically controllablelens 605 for magnification, clarification and the like. In addition, insome embodiments, the media projection unit 604 is detachable via adirect data coupling which is USB, Firewire®, high definition mediainterface (HDMI), a proprietary coupling, or the like. In someinstances, the audio sensor 608, the projector 604 and the visual sensor606 are combined into a single unit and detachable from the mainenclosure 602. In other instances, the audio sensor 608, the projector604 and the visual sensor 606 are separately detachable from the mainenclosure 602. In some embodiments, the audio sensor 608, the projector604 and the visual sensor 606 are attachable to the enclosure 602 inseveral configurations and several locations across the enclosure 602.

The visual sensor 606 captures images and video to store in memory andis also equipped with an ocular lens for magnification. According to oneembodiment, the visual sensor 606 may be positioned underneath theprojector, or above the projector. In one embodiment, the visual sensor606 must face the same direction as the media projection unit 604 inorder to accommodate touch gestures on the projection surface. In otherembodiments, however, the visual sensor 606 may be located on adifferent side of the enclosure 602 as that of the media projection unit604. The audio sensor 608 is similarly located on the same side ofenclosure 602 as the media projection unit 604 and the visual sensor606, although the present invention does not limit the audio sensor 604from being located on the adjacent or opposite side of the visual sensor606. The I/O reader 610 can be located on any side of the enclosure 602that is conveniently accessible by a user of the apparatus 600.

The enclosure 602 may also have a plurality of media projection units604, visual sensors 606, audio sensors 608 and I/O readers 610. In oneembodiment, the visual sensor 606 is a three-dimensional sensor and themedia projection unit 604 is capable of projecting three-dimensionalimages onto a projection surface.

The embodiments of the present invention may be embodied as methods,apparatus, electronic devices, and/or computer program products.Accordingly, the embodiments of the present invention may be embodied inhardware and/or in software (including firmware, resident software,micro-code, etc.), which may be generally referred to herein as a“circuit” or “module”. Furthermore, the present invention may take theform of a computer program product on a computer-usable orcomputer-readable storage medium having computer-usable orcomputer-readable program code embodied in the medium for use by or inconnection with an instruction execution system. In the context of thisdocument, a computer-usable or computer-readable medium may be anymedium that can contain, store, communicate, propagate, or transport theprogram for use by or in connection with the instruction executionsystem, apparatus, or device. These computer program instructions mayalso be stored in a computer-usable or computer-readable memory that maydirect a computer or other programmable data processing apparatus tofunction in a particular manner, such that the instructions stored inthe computer usable or computer-readable memory produce an article ofmanufacture including instructions that implement the function specifiedin the flowchart and/or block diagram block or blocks.

The computer-usable or computer-readable medium may be, for example butnot limited to, an electronic, magnetic, optical, electromagnetic,infrared, or semiconductor system, apparatus, device, or propagationmedium, but exclude transitory media such as transitory waves. Morespecific examples (a non-exhaustive list) of the computer-readablemedium include the following: hard disks, optical storage devices, atransmission media such as those supporting the Internet or an intranet,magnetic storage devices, an electrical connection having one or morewires, a portable computer diskette, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), an optical fiber, and a compact disc read-only memory(CD-ROM).

Computer program code for carrying out operations of the presentinvention may be written in an object oriented programming language,such as Java®, Smalltalk or C++, and the like. However, the computerprogram code for carrying out operations of the present invention mayalso be written in conventional procedural programming languages, suchas the “C” programming language and/or any other lower level assemblerlanguages. It will be further appreciated that the functionality of anyor all of the program modules may also be implemented using discretehardware components, one or more Application Specific IntegratedCircuits (ASICs), or programmed Digital Signal Processors ormicrocontrollers.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the present disclosure and its practical applications, tothereby enable others skilled in the art to best utilize the inventionand various embodiments with various modifications as may be suited tothe particular use contemplated.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

1. A robotic assistant comprising: a projector for projecting media on asurface; a sensor for sensing media; and a motion control module formoving the robotic assistant.
 2. The robotic assistant of claim 1,wherein the projector is detachable from an enclosure of the roboticassistant.
 3. The robotic assistant of claim 1, further comprising adata input module for receiving media for projecting.
 4. The roboticassistant of claim 1, wherein the media comprises audio and/or videodata.
 5. The robotic assistant of claim 4, further comprising an audiocapture module coupled to the sensor for interpreting the audio data asvoice commands.
 6. The robotic assistant of claim 5 further comprising acommand module for executing the voice commands to control at least oneof the projector and the motion control module.
 7. The robotic assistantof claim 1 further comprising a surface detection module for detecting asurface on which to display the media.
 8. The robotic assistant of claim1 wherein the motion control module comprises two sets of wheels.
 9. Therobotic assistant of claim 7 wherein the surface detection modulesearches for surfaces which have dimensions proportional to those of theprojected media.
 10. The robotic assistant of claim 9 wherein thesurface detection module commands the motion control module to relocatethe robotic assistant to a new location.
 11. A method of utilizing arobotic assistant comprising: projecting media on a surface; sensingmedia in a surrounding environment; and moving the robotic assistant.12. The method of claim 11, wherein the projector is detachable from anenclosure of the robotic assistant.
 13. The method of claim 11, furthercomprising a data input module for receiving media for projecting. 14.The method of claim 11, wherein the media comprises audio and/or videodata.
 15. The method of claim 14, further comprising an audio capturemodule coupled to the sensor for interpreting the audio data as voicecommands.
 16. The method of claim 15 further comprising a command modulefor executing the voice commands to control at least one of theprojecting and the moving.
 17. The method of claim 11 further comprisinga surface detection module for detecting a second surface on which todisplay the media.
 18. The method of claim 11 wherein the projectingcomprises one of 2D and 3D projecting.
 19. The method of claim 17wherein the surface detection module searches for surfaces which havedimensions proportional to those of the projected media.
 20. The methodof claim 19 wherein the surface detection module commands the motioncontrol to relocate the robotic assistant to a new location in responseto the searching.